Representing Affective Facial Expressions for Robots and Embodied Conversational Agents by Facial Landmarks

Affective robots and embodied conversational agents require convincing facial expressions to make them socially acceptable. To be able to virtually generate facial expressions, we need to investigate the relationship between technology and human perception of affective and social signals. Facial landmarks, the locations of the crucial parts of a face, are important for perception of the affective and social signals conveyed by facial expressions. The goal of our study is to determine to what extent facial landmarks are sufficient to represent facial expressions of emotions. If facial landmarks contain sufficient information, it should be possible to recognize and generate emotional expressions based on the landmarks only. This study focusses on the human recognition of emotional expressions from landmark sequences. Using face-analysis software, we digitally extracted landmark sequences from full-face videos of acted emotions. The landmark sequences were presented to sixteen participants who were instructed to classify the sequences according to the emotion represented. The results of the experiment revealed that participants were able to recognize the emotion reasonably well from the landmark sequences, suggesting that landmarks contain information about the expressed emotions. The implications of our findings for the virtual generation of facial expressions in robots and embodied conversational agents are discussed. We conclude by stating that landmarks provide a sufficient basis for the virtual generation of emotions in humanoid agents. Caixia Liu · Jaap Ham · Cees Midden Human-Technology Interaction Group, Department of Industrial Engineering and Innovation Sciences, Eindhoven University of Technology, Eindhoven, The Netherlands E-mail: {c.liu, j.r.c.ham, c.j.h.midden}@tue.nl Caixia Liu · Eric Postma · Bart Joosten · Martijn Goudbeek Tilburg center for Cognition and Communication, Tilburg University, Tilburg, The Netherlands E-mail: {e.o.postma, b.joosten, m.b.goudbeek}@tilburguniversity.edu